As both the workplace and our personal lives rely increasingly on digital documents and files, a persistent means of storing those objects is becoming increasingly important. The applications that are used on all types of computing devices are creating more and more digital content that is important to retain. For example, common office and multimedia applications create and consume data of various types and formats such as text documents, spreadsheets, images, audio, video, and application specific data. Increasingly, it is desirable to both store and repeatedly access this data from a number of different devices over an extended period of time. User expectations and application functionality require that this data be available from on-demand, near-line, and archival data storage sources.
As personal and mobile computing systems have matured into useful devices, the primary storage medium for application data is most commonly a magnetic fixed drive or “hard drive.” This types of storage device is typically either a permanent, installable, or a removable device and is available to application software for data storage. As a part of a local personal computer or mobile computer, this storage is typically multi-purpose storage that is used for the operating system, local application installation, and data storage. It is typically accessible to a single user or small number of users and tends to be a data storage “island” for which data can be copied or written, but is not easily shared. With special security configurations and/or dedicated server computers, such data can be shared on the local network among users that have access to the local network. However, such shared access does not typically extend in a robust and secure fashion on wide area networks such as the Internet.
Application storage on personal and mobile storage systems are convenient, but generally lack the reliability for either medium-term or long-term storage. In part, this lack of reliability extends from the fact that most storage only saves one copy of the application data. In some cases, technologies like RAID (Redundant Arrays of Inexpensive Disks) are used on local networks to provide some data reliability. In general, applications and users are left to manage their own data redundancy in order to provide a measure of data resilience and reliability.
Solutions known as Object Storage Systems (object storage systems or OSS) are beginning to emerge that provide a robust means of preserving data while making such data available to a wide area network such as the Internet. An example of such an Object Storage System is disclosed by Vermeulen, et al. in U.S. Pat. No. 8,185,497 (2012) and application U.S. App. 2013/0212165 (2013). Data is stored on these systems in the form of Objects. Objects are comprised of the data to be stored, metadata that describes storage and content attributes of such data, and an identifier that can be used to reference such data. While several major embodiments exist for Object Storage Systems, such systems tend to have a consistent functions and behaviors that a) expose as a simple set of web services that allow objects to be listed, saved, retrieved, updated, and deleted, b) allow objects to be placed in folders or buckets that group objects logically, c) provide for storage of more than one replica of the object in one or more physical locations, and d) provide scheduled object integrity checking to insure that the replicas are consistent and when found to be inconsistent, new replicas of an object are automatically created. In addition, some Object Storage System embodiments provide for a) access control lists for folders and/or objects based on granted permissions and parameter, b) intelligent location of objects based on data age, usage, importance, and/or modifications, c) multiple versions of the same object, d) granular control of placement of objects at one or more locations, and e) account control and billing based on storage, bandwidth, or other limits.
Historically, the storage of important documents and works are measured by tens, hundreds, and sometimes even thousands of years. While Object Storage Systems (OSS) are effective for short-term and medium-term object storage, they are not well suited to address the challenges of longer-term object storage, e.g. the storage of objects for more than ten years. Such Object Storage Systems are optimized for cost and convenience at the expense of long term resilience and security, typically measuring resilience only by storage system characteristics and omitting resilience factors related to geopolitical, economic, and organization characteristics. In particular, such Object Storage Systems are inadequate for longer-term storage because a) a single Object Storage System requires large quantities of node-to-node bandwidth therefore geo-redundant replicas of such a system tend to be located within miles instead of being diverse across a large geographical area such as a continent, b) such Object Storage Systems require all nodes to be powered-on and communicating thus consuming a large amount of energy in order to maintain consistent reliable storage of objects that are rarely accessed, c) such Object Storage Systems are generally located in the same geopolitical area and are subject to political and/or legal risks such as confiscation, regulation, or forced shutdown, d) such Object Storage Systems are subject to economic risks of the Object Storage System operator such as bankruptcy, profitability, unknown storage costs for users, and business plan obsolescence, e) such Object Storage Systems are subject to geographic risks where said Object Storage System is located such as natural disaster, climate effects, and resource constraints such as energy shortages, f) current embodiments of such Object Systems typically do not provide robust mechanisms for end-to-end user confidentiality, authentication, and security, and g) current embodiments of such Object Systems typically do not provide for confidentiality, authentication, and security that will survive the lifetime of the user and/or said Object Storage Systems. A storage system disclosed by Grove, et al. in U.S. Pat. No. 8,892,677 (2014) allows the consumer of a storage system to dynamically request a one of a plurality of hosted storage services as a storage target. However, this system does not address the compelling requirements of long term data storage and therefore does not significantly improve on the shortcoming of Object Storage Systems as recited. Metzer, et al. in U.S. Pat. No. 8,726,351 (2014) discloses an archive system that provides highly granular security access and workflow methods for paper and electronic governmental archives that must be “retained forever”, but focuses primarily on the means of access and verification germane to that particular application. In an attempt to address local and hosted storage synchronization, Velummylum, et al. in U.S. App. 2012/0254340 (2012) addresses what would appear to systems and methods that have been long in use for the caching of documents and files during synchronization between systems.
A Distributed Storage System (DSS) as disclosed herein embodies enhances object storage functionality to address these and other shortcomings of a standalone Object Storage Systems. Significant benefits of such a Distributed Storage System include a) adaptability to very low bandwidth connecting networks thus achieving eventual consistency between Object Storage Systems that are widely distributed around the globe and/or connected by low bandwidth networks, b) efficiency that enables very low energy consumption per storage unit by allowing secondary replicas of objects to reside in Object Storage Systems that remain powered-off at rest except for periodic validations and object management operations thus making possible practical renewable energy based storage systems, c) resilience strategy that mitigates risk posed by political and legal frameworks such as confiscation, regulation, or forced shutdown by making possible operations across diverse political and legal environments, d) resilience strategy that mitigates risk posed by economic factors of the Object Storage System operator such as bankruptcy, profitability, unknown storage costs for users, and business plan obsolescence by making possible operations across a plurality of operating organizations, e) resilience strategy that mitigates risk posed by geographic factors such as natural disaster, climate effects, and resource constraints (e.g. energy shortages) by making possible operations that are located across a wide geographic area, f) strong endpoint to endpoint authentication and privacy functionality that is distributed across multiple political, geographic, and organizational boundaries such that the ownership and disclosure rights of stored objects are not ultimately compromised or waived, and g) structured and inheritable rights for user access, authentication, and privacy to stored objects whereby enabling multi-generational ownership of stored objects for tens and hundreds of years.